Systems, methods, and computer-readable media for generating computer-mediated reality display data

ABSTRACT

Systems, methods, and computer-readable media are provided for generating computer-mediated reality display data based on user instantaneous motion data.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of prior filed U.S. ProvisionalPatent Application No. 61/842,864, filed Jul. 3, 2013, which is herebyincorporated by reference herein in its entirety.

TECHNICAL FIELD

This can relate to systems, methods, and computer-readable media forgenerating computer-mediated reality display data and, moreparticularly, to systems, methods, and computer-readable media forgenerating computer-mediated reality display data based on userinstantaneous motion data.

BACKGROUND

Conventional computer-mediated reality systems (e.g., virtual realitysystems and augmented reality systems) have been developed in which auser is provided with a head mounted display. When the user moves his orher head around, the head mounted display changes the images beingdisplayed to the user based on the new position (e.g., reality) of thehead mounted display. Conventional head mounted display systems have atendency to induce dizziness or excessive eye fatigue on the user. Thesenegative effects can become even more pronounced if the user rapidlymoves his or her head. These negative effects can be caused by the headmounted display system's inability to present images to the user in amanner that is commensurate with the user's head movements. That is,when the user's head movements and images being displayed in responsethereto are out of sync, the user may not be presented with the bestvirtual reality or augmented reality experience. Accordingly, improveddisplay data generation for computer-mediated reality systems andmethods are needed.

SUMMARY

Systems, methods, and computer-readable media for generatingcomputer-mediated reality display data are provided.

In some embodiments, there is provided a computer-mediated realitysystem that may include at least one sensor, a mediated reality datasource, and a mediated reality display generator. The mediated realitydisplay generator may generate displayable mediated reality scene databased on current reality data of the system from the at least onesensor, mediated reality data from the mediated reality data source, andinstantaneous motion data of the system from the at least one sensor.For example, the current reality data of the system may be provided tothe mediated reality display generator from a first sensor of the atleast one sensor, and the instantaneous motion data of the system may beprovided to the mediated reality display generator from a second sensorof the at least one sensor. The mediated reality display generator maybe configured to generate displayable mediated reality frame data basedon the current reality data and the mediated reality data, and themediated reality display generator may be configured to generate thedisplayable mediated reality scene data by selecting a portion of thedisplayable mediated reality frame data based on the instantaneousmotion data. In some embodiments, a center of the selected portion maybe offset from a center of the displayable mediated reality frame databased on the instantaneous motion data. The current reality data of thesystem may be indicative of the current location of the system, wheresuch current reality data may be provided to the mediated realitydisplay generator from a global positioning system sensor, and/or wherethe mediated reality data of the system may be indicative of informationthat is both related to the current location of the system and useful toa user of the system. The current reality data of the system may beindicative of the current orientation of the system, where such currentreality data may be provided to the mediated reality display generatorfrom at least one of a compass sensor and a gyroscope sensor. Theinstantaneous motion data of the system may be provided to the mediatedreality display generator from at least one accelerometer sensor.

In other embodiments, there is provided a method that may includedetermining a current reality of a computer-mediated reality displaysystem, determining a current instantaneous motion of thecomputer-mediated reality display system, and generating displayablemediated reality scene data based on the determined current reality andthe determined current instantaneous motion. In some embodiments, such amethod may also include generating displayable mediated reality framedata based on the determined current reality, and selecting a portion ofthe displayable mediated reality frame data as the displayable mediatedreality scene data based on the determined current instantaneous motion.A center of the selected portion may be offset from a center of thedisplayable mediated reality frame data based on the determined currentinstantaneous motion. In some other embodiments, such a method mayinclude identifying displayable mediated reality frame data based on thedetermined current reality and selecting a field of view portion of theidentified displayable mediated reality frame data as the displayablemediated reality scene data, where the position of the selected field ofview portion of the identified displayable mediated reality frame datawith respect to the entirety of the identified displayable mediatedreality frame data may be a first position that is based on thedetermined current instantaneous motion. Such a method may be repeatedat a later second time, in which the second position of the selectedsecond field of view portion of the identified second displayablemediated reality frame data with respect to the entirety of theidentified second displayable mediated reality frame data may bedifferent than the first position at the earlier first time. In suchembodiments, the first position may be different than the secondposition with respect to two axes.

In yet other embodiments, there is provided a non-transitorycomputer-readable medium that may include computer-readable instructionsrecorded thereon for displaying mediated reality scene data, determininga current reality of a computer-mediated reality display system,determining a current instantaneous motion of the computer-mediatedreality display system, and virtually altering the frame rate of thedisplayed mediated reality scene data by moving the field of view of themediated reality scene data with respect to the determined currentreality of the system based on the determined current instantaneousmotion. The determined current instantaneous motion may include aderivative of the current acceleration of the computer-mediated realitydisplay system. In such embodiments, the moving may include moving thefield of view with respect to the determined current reality of thesystem in a first particular direction when the current acceleration ispositive in the first particular direction, and/or moving the field ofview with respect to the determined current reality of the system in asecond particular direction opposite to the first particular directionwhen the current acceleration is negative in the first particulardirection.

In still yet other embodiments, there is provided a machine-readablestorage medium that may provide instructions that, when executed by aprocessor, may cause the processor to perform operations. The operationsmay include determining a current reality of a computer-mediated realitydisplay system, determining a current instantaneous motion of thecomputer-mediated reality display system, and generating displayablemediated reality scene data based on the determined current reality andthe determined current instantaneous motion.

This Summary is provided merely to summarize some example embodiments,so as to provide a basic understanding of some aspects of the subjectmatter described in this document. Accordingly, it will be appreciatedthat the features described in this Summary are merely examples andshould not be construed to narrow the scope or spirit of the subjectmatter described herein in any way. Other features, aspects, andadvantages of the subject matter described herein will become apparentfrom the following Detailed Description, Figures, and Claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects of the disclosure, its nature, and variousfeatures will become more apparent upon consideration of the followingdetailed description, taken in conjunction with the accompanyingdrawings, in which like reference characters refer to like partsthroughout, and in which:

FIG. 1 is a schematic view of an illustrative system for generatingcomputer-mediated reality display data, in accordance with variousembodiments;

FIG. 2 is a schematic view of illustrative portions of the system ofFIG. 1, in accordance with various embodiments;

FIG. 3 is an exemplary graph plotting system velocity versus systemreality, in accordance with various embodiments;

FIG. 4 is an exemplary diagram illustrating a system's movement inreality based on the graph of FIG. 3, in accordance with variousembodiments;

FIGS. 5A-5E are exemplary computer-mediated reality display frame dataand associated screen data based on the graph of FIG. 3, in accordancewith various embodiments; and

FIG. 6 is a flowchart of an illustrative process for generatingcomputer-mediated reality display data, in accordance with variousembodiments.

DETAILED DESCRIPTION

Systems, methods, and computer-readable media for generatingcomputer-mediated reality display data are provided and described withreference to FIGS. 1-6.

A computer-mediated reality system may include a mediated reality framegenerator (e.g., a graphics processing unit) that may generatedisplayable frame data based on a current reality of the system andmediated reality data associated with that current reality. For example,the current reality of the system may be determined using any suitablesensor or combination of sensors and may be indicative of the currentlocation of the system (e.g., via a GPS sensor), the current orientationof the system (e.g., via a compass sensor), and/or any other suitableinformation that may define the current relationship of the system withrespect to its environment at a particular moment in time, while themediated reality data may be any suitable data not naturally present inthe system's environment but that may be associated with the system'senvironment in any suitable way (e.g., augmented or virtual reality dataor video game data). The system may also include a mediated realityscene generator (e.g., a field of view (“FOV”) processing unit) that maygenerate displayable scene data based on the generated displayable framedata and a current instantaneous motion of the system. For example, thecurrent instantaneous motion of the system may be determined using anysuitable sensor or combination of sensors and may be indicative of theinstantaneous velocity and/or the instantaneous acceleration of thesystem in one or more directions (e.g., via one or more accelerometersensors) at a particular moment in time (e.g., at the same moment intime as the current reality of the system was determined for thegenerated displayable frame data). The mediated reality scene generatormay generate the displayable scene data by selecting a particularportion of the generated displayable frame data as the displayable scenedata based on the current instantaneous motion of the system.

For example, if the current instantaneous motion is indicative of thesystem accelerating in a first direction along a first axis, the centerof the selected particular portion of the generated displayable framedata may be offset from the center of the entirety of the generateddisplayable frame data in that first direction along that first axis.Alternatively or additionally, if the current instantaneous motion isindicative of the system decelerating in that first direction along thatfirst axis, the center of the selected particular portion of thegenerated displayable frame data may be offset from the center of theentirety of the generated displayable frame data in a second directionopposite that first direction along that first axis. Therefore, such asystem may virtually alter the frame rate of mediated reality dataprovided to a user of the system (e.g., as displayable scene data via adisplay of the system) by moving a field of view of the displayablescene data with respect to an orientation point (e.g., the center) ofthe displayable frame data based on a current instantaneous motion ofthe system. This may reduce the negative effects (e.g., dizziness, eyefatigue, etc.) that may often be associated with a system's presentationof mediated reality data that is constantly updated based on the currentreality of the system when the system is being moved rapidly.

FIG. 1 is a schematic view of an illustrative system 100 for generatingcomputer-mediated reality display data in accordance with someembodiments of the invention. For example, in some embodiments, as shownin FIG. 1, system 100 may include, but is not limited to, a musicplayer, video player, still image player, game player, other mediaplayer and/or recorder, cellular telephone, other wireless communicationdevice, personal digital assistant, remote control, pager, computer(e.g., a desktop, laptop, tablet, server, etc.), monitor, television,stereo equipment, set up box, set-top box, or any combination thereofthat may be capable of generating computer-mediated reality data. System100 may be any portable, mobile, hand-held, or miniature electronicdevice that may be configured to generate computer-mediated reality datawherever a user travels. Some miniature electronic devices may have aform factor that is smaller than that of hand-held electronic devices.Illustrative miniature electronic devices can be integrated into variousobjects that may include, but are not limited to, watches, headsets,virtual reality devices, glasses, other wearable electronics,accessories for sporting equipment, accessories for fitness equipment,or any combination thereof. System 100 may include a single integralelectronic device or multiple electronic components that may be usedtogether to form an electronic system.

As shown in FIG. 1, for example, system 100 may include a processor 102,memory 104, communications component 106, power supply 108, inputcomponent 110, output component 112, and sensor 114. System 100 may alsoinclude a bus 115 that may provide one or more wired or wirelesscommunications links or paths for transferring data and/or power to,from, or between various other components of system 100. In someembodiments, one or more components of system 100 may be combined oromitted. Moreover, system 100 may include other components not combinedor included in FIG. 1 and/or several instances of the components shownin FIG. 1. For the sake of simplicity, only one of each of thecomponents of system 100 is shown in FIG. 1.

Memory 104 of system 100 may include one or more storage mediums,including for example, a hard-drive, flash memory, permanent memory suchas read-only memory (“ROM”), semi-permanent memory such as random accessmemory (“RAM”), any other suitable type of storage component, or anycombination thereof. Memory 104 may include cache memory, which may beone or more different types of memory used for temporarily storing datafor electronic device applications. Memory 104 may store media data(e.g., audio, image, and/or video files), software (e.g., forimplementing functions on system 100), firmware, computer-mediatedreality data information, preference information (e.g., media playbackpreferences), wireless connection information (e.g., information thatmay enable system 100 to establish a wireless connection), subscriptioninformation (e.g., information that keeps track of podcasts ortelevision shows or other media a user subscribes to), contactinformation (e.g., telephone numbers and e-mail addresses), calendarinformation, any other suitable data, or any combination thereof.

Communications component 106 of system 100 may be provided to allowsystem 100 to communicate with one or more other electronic systems,electronic devices, or servers using any suitable wired or wirelesscommunications protocol. For example, first communications component 106may support Wi-Fi (e.g., an 802.11 protocol), ZigBee (e.g., an 802.15.4protocol), WiDi™, Ethernet, Bluetooth™, Bluetooth™ Low Energy (“BLE”),high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHzcommunication systems), infrared, transmission control protocol/internetprotocol (“TCP/IP”) (e.g., any of the protocols used in each of theTCP/IP layers), Stream Control Transmission Protocol (“SCTP”), DynamicHost Configuration Protocol (“DHCP”), hypertext transfer protocol(“HTTP”), BitTorrent™, file transfer protocol (“FTP”), real-timetransport protocol (“RTP”), real-time streaming protocol (“RTSP”),real-time control protocol (“RTCP”), Remote Audio Output Protocol(“RAOP”), Real Data Transport Protocol™ (“RDTP”), User Datagram Protocol(“UDP”), secure shell protocol (“SSH”), wireless distribution system(“WDS”) bridging, any communications protocol that may be used bywireless and cellular telephones and personal e-mail devices (e.g.,Global System for Mobile Communications (“GSM”), GSM plus Enhanced Datarates for GSM Evolution (“EDGE”), Code Division Multiple Access(“CDMA”), Orthogonal Frequency-Division Multiple Access (“OFDMA”), highspeed packet access (“HSPA”), multi-band, etc.), any communicationsprotocol that may be used by a low power Wireless Personal Area Network(“6LoWPAN”) module, any other communications protocol, or anycombination thereof. Communications component 106 may be configured toenable system 100 to be electrically coupled to one or more otherelectronic systems, electronic devices, or servers and to communicatewith that other entity, either wirelessly or via a wired connection.

Power supply 108 of system 100 may provide power to one or more of thecomponents of system 100. In some embodiments, power supply 108 can becoupled to a power grid (e.g., when system 100 is not a portable device,such as a desktop computer). In some embodiments, power supply 108 caninclude one or more batteries for providing power (e.g., when system 100is a portable device, such as a wearable computer, such as ahead-mounted display-based computing device, such as Google Glass™. Asanother example, power supply 108 can be configured to generate powerfrom a natural source (e.g., solar power using solar cells).

One or more input components 110 of system 100 may be provided to permita user to interact or interface with system 100. For example, inputcomponent 110 can take a variety of forms, including, but not limitedto, a touch pad, dial, click wheel, scroll wheel, touch screen, one ormore buttons (e.g., a keyboard), mouse, joy stick, track ball,microphone, camera, motion sensor, proximity sensor, light detector, andcombinations thereof. Each input component 110 can be configured toprovide one or more dedicated control functions for making selections orissuing commands associated with operating system 100.

System 100 may also include one or more output components 112 that maypresent information (e.g., visual, audible, and/or tactile information)to a user of system 100. Output component 112 of system 100 may takevarious forms, including, but not limited to, audio speakers,headphones, audio lines-out, visual displays, video lines-out, antennas,infrared ports, rumblers, vibrators, or combinations thereof.

For example, as shown in FIG. 1 and described in more detailhereinbelow, system 100 may include a display output component 112 as anoutput component. Such a display output component 112 may include anysuitable type of output interface for presenting visual data (e.g.,computer-mediated reality display data) to a user. In some embodiments,display output component 112 may include any suitable type of visualdisplay, including, but not limited to, a liquid crystal display(“LCD”), a light emitting diode (“LED”) display, an organiclight-emitting diode (“OLED”) display, a surface-conductionelectron-emitter display (“SED”), a carbon nanotube display, ananocrystal display, any other suitable type of display, or combinationthereof. Alternatively, such a display output component 112 can includea movable display or a projecting system for providing a display ofcontent on a surface remote from system 100, such as, for example, avideo projector, a head-up display, or a three-dimensional (e.g.,holographic) display. As another example, such a display outputcomponent 112 may include a digital or mechanical viewfinder, such as aviewfinder of the type found in compact digital cameras, reflex cameras,or any other suitable still or video camera. Display output component112 can be operative to output the playback of media content and/orcomputer-mediated reality display data that may be under the directionof processor 102 to an environment external to system 100 for receipt bya user.

It should be noted that one or more input components 110 and one or moreoutput components 112 may sometimes be referred to collectively hereinas an input/output (“I/O”) component or I/O interface (e.g., inputcomponent 110 and display 112 as an I/O component or I/O interface). Forexample, input component 110 and display output component 112 maysometimes be a single I/O component, such as a touch screen assembly,that may receive input information through a user's touch of a displayscreen assembly and that may also provide visual information to a uservia that same display screen assembly.

Sensor 114 of system 100 may include any suitable sensor that may beoperative to determine the location, orientation, position,instantaneous motion, and/or any other suitable characteristic of system100 within its environment and/or any suitable characteristic of theenvironment itself. For example, sensor 114 may be one or more of acompass, accelerometer, gyroscope, global positioning system (“GPS”),microphone, camera, motion sensor, proximity sensor, light detector, anycombination thereof, and/or any other suitable sensor.

Processor 102 of system 100 may include any processing circuitry thatmay be operative to control the operations and/or performance of one ormore components of system 100. For example, processor 102 may receiveinput signals from input component 110 and/or drive output signalsthrough output component 112. In some embodiments, as shown in FIG. 1,processor 102 may be used to run one or more applications, such as anapplication 103. Application 103 may include, but is not limited to, oneor more operating system applications, firmware applications,computer-mediated reality data applications, media playbackapplications, media editing applications, or any other suitableapplications. For example, processor 102 may load application 103 as acomputer-mediated reality interface program to determine howinstructions or data received via an input component 110 or othercomponent of device 100 (e.g., one or more sensors 114) may manipulatethe way in which information is stored and/or provided to the user viaan output component 112 (e.g., via a display output component 112).Application 103 may be accessed by processor 102 from any suitablesource, such as from memory 104 (e.g., via bus 115), from any othersuitable remote system or electronic device (e.g., via communicationscomponent 106), or from any other suitable source.

System 100 may also be provided with a housing 101 that may at leastpartially enclose one or more of the components of system 100 forprotection from debris and other degrading forces external to system100. In some embodiments, one or more of the components of system 100may be provided within its own housing (e.g., input component 110 may bea joystick within its own housing that may wirelessly or through a wirecommunicate with processor 102 and output component 112, which may beprovided within their own housing). System 100 may be configured invarious ways and may include various combinations of various componentswhile still generating computer-mediated reality display data based onuser instantaneous motion data.

FIG. 2 shows a schematic view of a computer-mediated reality displaydata generator subsystem 201 of system 100 that may be provided to atleast partially generate computer-mediated reality display data based onuser instantaneous motion data. For example, in some embodiments,mediated reality display data generator subsystem 201 may include amediated reality frame generator 202 and a mediated reality scenegenerator 208. Frame generator 202 may be configured to receive mediatedreality data 211 from a mediated reality data source 290 and/or framedefinition data 213 from at least one sensor 114, and then framegenerator 202 may be configured to generate displayable frame data 215based on received mediated reality data 211 and received framedefinition data 213. For example, at least one sensor 114 may beconfigured to generate frame definition data 213, which may beindicative of the current location of system 100, position of system100, orientation of system 100, and/or any other suitable data that mayin one or more ways define system 100's current relationship with theenvironment (e.g., a user's current reality or relationship with theenvironment when system 100 is held, worn, or otherwise used by theuser). For example, system 100 may be a wearable computer with ahead-mounted display (e.g., Google Glass™, and frame definition data 213may be data indicative of the current location (e.g., GPS sensor data),the current position and/or current orientation (e.g., compass sensordata and/or gyroscope sensor data), and/or any other suitable type ofrelationship data that may be indicative of the current relationship ofthe head-mounted display of system 100 with respect to its environment.When a user wearing such a head-mounted display moves to a new locationand/or alters the orientation of the head-mounted display, framedefinition data 213 may be updated to appropriately describe the currentreality or existence of the head-mounted display within its surroundingenvironment (e.g., the current reality of system 100 and its user).

Frame generator 202 may receive such frame definition data 213 as wellas any accessible mediated reality data 211 from mediated reality datasource 290 in order to define and/or generate displayable frame data 215for that current existence/reality of system 100. Data source 290 may beany suitable data source accessible to subsystem 201 of system 100 thatmay provide suitable mediated reality data 211 that may be useful to auser of system 100 for its current reality. Such mediated reality data211 may include any suitable data, such as visual data, that may beoverlaid on a user's current reality to add value to the user'sexperience (e.g., street name mediated reality data may be overlaid on auser's reality view of an intersection with no street signs).Additionally or alternatively, such mediated reality data 211 mayinclude video game data, virtual reality data, and/or augmented realityexperience data that may be based on the current location and/orposition and/or orientation of the system 100, which may be determinedby data 213. Data source 290 may be any suitable data source accessibleto subsystem 201, such as memory 104 or a remote server accessible tosystem 100 via communications component 106. Based on the user's currentreality (e.g., as defined by frame definition data 213) and based onavailable mediated reality data 211 (e.g., as provided by data source290), frame generator 202 may generate appropriate displayable framedata 215 that may be provided to a user (e.g., via display outputcomponent 112) for enhancing the user's experience of his or her currentreality. Frame generator 202 may be a graphics processing unit (“GPU”)or any other suitable component or combination of components and/ormodules.

In addition to leveraging the user's current reality (e.g., currentposition, location, orientation, etc.) using frame definition data 213to determine what mediated reality data may be provided to the user,subsystem 201 may also leverage the instantaneous motion information ofthe user at its current reality (e.g., the instantaneous velocity and/orthe instantaneous acceleration of system 100 in one or more directionsfor the user's reality at a particular moment in time). For example, atleast one sensor 114 may generate scene definition data 217 that may beindicative of one or more characteristics of instantaneous motion ofsystem 100 (e.g., a head-mounted display), such as via one or moreaccelerometers or any other suitable type of sensor. By taking intoaccount such instantaneous motion, subsystem 201 may adjust whatmediated reality data is provided to the user in order to minimizeirritation (e.g., headaches) that may be caused by updating mediatedreality data to match the rate at which a user's reality is updated. Forexample, mediated reality scene generator 208 may receive displayableframe data 215 from frame generator 202 and scene definition data 217from one or more sensors 114, and scene generator 208 may then generatedisplayable scene data 219 by utilizing a specific portion of thereceived displayable frame data 215, where that specific portion may bechosen by scene generator 208 based on the received scene definitiondata 217. Such generated displayable scene data 219 may then be providedby scene generator 208 to display output component 112 for use by a userof system 100. Scene generator 208 may be a field of view (“FOV”)processing unit or any other suitable component or combination ofcomponents and/or modules.

For example, as shown in FIG. 3 by a graph 300 and in FIG. 4 by adiagram 400, a user U may wear or otherwise hold or manipulate system100 such that its reality or existence (e.g., as may be determined byframe definition data 213) may change from a first reality A at a firsttime to another reality B at a second time after the first time toanother reality C at a third time after the second time to anotherreality D at a fourth time after the third time and to yet anotherreality E at a fifth time after the fourth time. The change between anytwo consecutive ones of these realities may differ from each other. Forexample, the manner in which system 100 may be moved from reality A toreality B may be quite different than the manner in which system 100 maybe moved from reality B to reality C. As just one example, as shown inFIG. 3 by graph 300, system 100 may start at reality A with no movementand then may reach a velocity of value 1 when it gets to reality B, andsystem 100 may achieve an even higher velocity of value 2 when it movesfrom reality B to reality C (e.g., when a user U quickly swivels itshead to advance its focus along the X-axis). Then, system 100 may slowback down to the velocity of value 1 when it moves from reality C toreality D, and then system 100 may return to a state of rest when itmoves from reality D to reality E. While frame definition data 213 maynot be indicative of the manner in which system 100 moved when changingbetween realities (e.g., frame generator 202 may not take into accountthe type of movement utilized by system 100 to achieve a new realitywhen frame generator 202 generates displayable frame data 215), scenedefinition data 217 may be indicative of such movement (e.g., scenegenerator 208 may take into account the manner in which system 100 wasmoving when system 100 achieved a new reality such that scene generator208 may generate displayable scene data 219 accordingly). As shown, theinstantaneous motion (e.g., directional instantaneous velocity) ofsystem 100 when at reality A may be 0, while it may be +1 at reality B,+2 at reality C, −1 at reality D, and 0 at reality E, and scenegenerator 208 may receive that instantaneous motion data (e.g., as scenedefinition data 217) for a particular reality and scene generator 208may use that motion data to select a specific portion of displayableframe data 215 to be provided as displayable scene data 219 to the userwhen system 100 is at that respective reality.

For example, as shown in FIG. 5A, when system 100 is at reality A,generator subsystem 201 may process data 500A, which may includedisplayable frame data 515 a (e.g., frame data 215 generated bygenerator 202 based on frame definition data 213 and mediated realitydata 211 when system 100 is at reality A) and displayable scene data 519a (e.g., scene data 219 generated by generator 208 based on scenedefinition data 217 and frame data 215 (e.g., frame data 515 a) whensystem 100 is at reality A). In some embodiments, scene data 519 a maybe a specific selected portion of frame data 515 a. For example, theshape of scene data 519 a may be the same as the shape of frame data 515a, but only a portion of the size (e.g., scene data may be a specific20% portion of the generated frame data, but may have the samerectangular geometry). However, which specific portion of frame data 515a may be used as scene data 519 a may be determined using scenedefinition data 217. For example, as shown in FIG. 5A, the center offrame data 515 a and the center of scene data 519 a may be the same(e.g., 0 displacement therebetween) due to scene definition data 217 atreality A being indicative of 0 instantaneous motion. However, as shownin FIG. 5B for data 500B when system 100 is at reality B, the center ofscene data 519 b may be offset from the center of frame data 515 b by a+1 amount (e.g., along the X-axis) due to scene definition data 217 atreality B being indicative of +1 instantaneous motion. Following thesame example, as shown in FIG. 5C for data 500C when system 100 is atreality C, the center of scene data 519 c may be offset from the centerof frame data 515 c by a +2 amount (e.g., along the X-axis) due to scenedefinition data 217 at reality C being indicative of +2 instantaneousmotion. Moreover, as shown in FIG. 5D for data 500D when system 100 isat reality D, the center of scene data 519 d may be offset from thecenter of frame data 515 d by a −1 amount (e.g., along the X-axis) dueto scene definition data 217 at reality D being indicative of −1instantaneous motion. Moreover, as shown in FIG. 5E for data 500E whensystem 100 is at reality E, the center of scene data 519 e may be thesame as the center of frame data 515 e (e.g., 0 displacementtherebetween) due to scene definition data 217 at reality E beingindicative of 0 instantaneous motion.

Therefore, this may allow subsystem 201 to virtually alter the framerate of mediated reality data provided to the user (e.g., as displayablescene data 219 via display output component 112) by moving displayablescene data 219 with respect to an orientation point (e.g., the center)of displayable frame data 215. For example, this may allow subsystem 201to virtually speed up the frame rate of mediated reality data providedto the user by moving the center of displayable scene data 219 ahead ofthe center of displayable frame data 215 when the instantaneous motionfor system 100 at the particular reality is positive (e.g., as shown inFIGS. 5B and 5C when system 100 has a positive instantaneous velocity orinstantaneous acceleration in the +X-direction). Additionally oralternatively, this may allow subsystem 201 to virtually slow down theframe rate of mediated reality data provided to the user by moving thecenter of displayable scene data 219 behind the center of displayableframe data 215 when the instantaneous motion for system 100 at theparticular reality is negative (e.g., as shown in FIG. 5D when system100 has a negative instantaneous velocity or instantaneous accelerationin the +X-direction). For example, while FIG. 4 may show a user Urotating its head to advance a focus of system 100 along a positivedirection of the X-axis in an X-Z plane from reality A to reality E(e.g., by changing orientational reality of system 100 worn on the headof user U, which may change the pointing direction of a camera of system100), system 100 may provide each one of displayable scene data 519a-519 e on display output component 112 in an X-Y plane, such thatchanges to the instantaneous motion of system 100 along the X-axis ofFIG. 4 (e.g., as may be indicated by scene definition data 217) mayaffect how the selected portion of displayable frame data 515 asdisplayable scene data 519 may be positioned with respect to theentirety of displayable frame data 515 along the X-axis of FIGS. 5A-5E.

Although FIGS. 5A-5E may only show the center of scene data 219 (e.g.,scene data 519 a-519 e) offset from the center of frame data 215 (e.g.,frame data 515 a-515 e) only along one axis (e.g., along the X-axis),scene data 219 may be offset from frame data 215 along two or more axes,which may be due to instantaneous motion of system 100 as measured alongtwo different axes (e.g., the instantaneous velocity of system 100 asmeasured along each of two axes by each of two accelerometers at aspecific reality of system 100). Additionally or alternatively, althoughFIGS. 5A-5E may only show the center of scene data 219 (e.g., scene data519 a-519 e) offset from the center of frame data 215 (e.g., frame data515 a-515 e) based on scene definition data 217, any suitable referencepoint of scene data 219 may be offset from any suitable reference pointof frame data 215 based on scene definition data 217. Moreover, in someembodiments, generators 202 and 208 may be combined (e.g., as mediatedreality display generator 206 of FIG. 2), such that frame definitiondata 213, scene definition data 217, and mediated reality data 211 maybe leveraged all at once to generate scene data 219 for delivery to auser via display output component 112. For example, rather thansubsystem 201 configuring generator 202 to generate the entirety offrame data 215 (e.g., based on frame definition data 213 and realitydata 211), and rather than configuring generator 208 to then select aparticular portion of that entire frame data 215 to use as scene data219 (e.g., based on scene definition data 217), generator 206 ofsubsystem 201 may be configured to only generate that particular portionof frame data 215 for use as scene data 219.

There may be various suitable ways to determine which portion ofdisplayable frame data for a particular reality may be selected for useas displayable scene data for that particular reality (e.g., usinginstantaneous motion information for that particular reality). Forexample, the position of the selected scene data with respect to theframe data for a particular reality may be related (e.g., directlyproportional) to the acceleration of the system along one or more axesat that particular reality, and may also be related to the position ofthe selected scene data with respect to the frame data for the previousreality occurring just before that particular reality. Alternatively oradditionally, the position of the selected scene data with respect tothe frame data for a particular reality may be at least partially basedon a derivative of the acceleration of the system at the instance oftime of that particular reality. The detected instantaneous motion ofthe system at a particular reality may be utilized as a reference forwhat delta in space the next displayable scene data may fall withrespect to the origin of the displayable frame data for that particularreality. This may allow the system to virtually speed up the frame rateof mediated reality data provided to the user (e.g., via display outputcomponent 112) by moving the center of the displayable scene data aheadof the center of the displayable frame data in a particular directionwhen the instantaneous motion for the system at the particular realityis positive with respect to that particular direction, and may virtuallyslow down the frame rate of mediated reality data provided to the userby moving the center of the displayable scene data behind the center ofthe displayable frame data in a particular direction when theinstantaneous motion for the system at the particular reality isnegative with respect to that particular direction.

FIG. 6 is a flowchart of an illustrative process 600 for generatingcomputer-mediated reality display data based on instantaneous motiondata. At step 602, process 600 may include determining a current realityof a computer-mediated reality display system. For example, subsystem201 (e.g., generator 202) may determine the current reality of system100 (e.g., using frame definition data 213). At step 604, process 600may include determining a current instantaneous motion of thecomputer-mediated reality display system. For example, subsystem 201(e.g., generator 208) may determine the current instantaneous motion ofsystem 100 (e.g., using scene definition data 217). At step 606, process600 may include generating displayable mediated reality scene data basedon the current reality and the current instantaneous motion. Forexample, subsystem 201 (e.g., generator 208) may generate displayablescene data 219, which may be based on the current reality of system 100(e.g., frame definition data 213 and any accessible mediated realitydata (e.g., mediated reality data 211)), and on the currentinstantaneous motion of system 100 (e.g., scene definition data 217). Insome embodiments, process 600 may also include generating displayablemediated reality frame data (e.g., displayable frame data 215) based onthe current reality. Moreover, process 600 may also include selecting aportion of the displayable mediated reality frame data as thedisplayable mediated reality scene data based on the instantaneousmotion. In some embodiments, a center or other point of origin of theselected portion of the displayable mediated reality frame data may beoffset from a center or other point of origin of the displayablemediated reality frame data based on the instantaneous motion.

It is to be understood that the steps shown in process 600 of FIG. 6 ismerely illustrative and that existing steps may be modified or omitted,additional steps may be added, and the order of certain steps may bealtered.

Moreover, one, some, or all of the processes described with respect toFIGS. 1-6, as well as any other aspects of the disclosure, may each beimplemented by software, but may also be implemented in hardware,firmware, or any combination of software, hardware, and firmware.Instructions for performing these processes may also be embodied asmachine- or computer-readable code recorded on a machine- orcomputer-readable medium. In some embodiments, the computer-readablemedium may be a non-transitory computer-readable medium. Examples ofsuch a non-transitory computer-readable medium include but are notlimited to a read-only memory, a random-access memory, a flash memory, aCD-ROM, a DVD, a magnetic tape, a removable memory card, and a datastorage device (e.g., memory 104 of FIG. 1). In other embodiments, thecomputer-readable medium may be a transitory computer-readable medium.In such embodiments, the transitory computer-readable medium can bedistributed over network-coupled computer systems so that thecomputer-readable code is stored and executed in a distributed fashion.For example, such a transitory computer-readable medium may becommunicated from one electronic device to another electronic deviceusing any suitable communications protocol (e.g., the computer-readablemedium may be communicated to system 100 from a remote server viacommunications component 106 (e.g., as at least a portion of anapplication 103)). Such a transitory computer-readable medium may embodycomputer-readable code, instructions, data structures, program modules,or other data in a modulated data signal, such as a carrier wave orother transport mechanism, and may include any information deliverymedia. A modulated data signal may be a signal that has one or more ofits characteristics set or changed in such a manner as to encodeinformation in the signal.

It is to be understood that any or each module of subsystem 201 (e.g.,generator modules 202, 206, and/or 208) may be provided as a softwareconstruct, firmware construct, one or more hardware components, or acombination thereof. For example, subsystem 201 may be described in thegeneral context of computer-executable instructions, such as programmodules, that may be executed by one or more computers or other devices.Generally, a program module may include one or more routines, programs,objects, components, and/or data structures that may perform one or moreparticular tasks or that may implement one or more particular abstractdata types. It is also to be understood that the number, configuration,functionality, and interconnection of the modules of subsystem 201 aremerely illustrative, and that the number, configuration, functionality,and interconnection of existing modules may be modified or omitted,additional modules may be added, and the interconnection of certainmodules may be altered.

At least a portion of one or more of the modules of subsystem 201 may bestored in or otherwise accessible to system 100 in any suitable manner(e.g., in memory 104 of system 100). Any or each module of subsystem 201may be implemented using any suitable technologies (e.g., as one or moreintegrated circuit devices), and different modules may or may not beidentical in structure, capabilities, and operation. Any or all of themodules of subsystem 201 may be mounted on an expansion card, mounteddirectly on a system motherboard, or integrated into a system chipsetcomponent (e.g., into a “north bridge” chip). Subsystem 201 may includeany amount of dedicated media playback memory, may include no dedicatedmedia playback memory and may rely on device memory or network memory(e.g., memory of a remote server), or may use any combination thereof.

Subsystem 201 may be a dedicated system that may be implemented usingone or more expansion cards adapted for various bus standards. Forexample, all of the modules of subsystem 201 may be mounted on differentinterconnected expansion cards or all of the modules of a system may bemounted on one expansion card. The modules of subsystem 201 mayinterface with a motherboard or processor 102 of system 100 through anexpansion slot (e.g., a peripheral component interconnect (“PCI”) slotor a PCI express slot). Alternatively, subsystem 201 need not beremovable but may include one or more dedicated modules that may includememory (e.g., RAM) dedicated to the utilization of the module. In otherembodiments, subsystem 201 may be a system integrated into system 100.For example, a module of subsystem 201 may utilize a portion of devicememory 104 of system 100. One or more of the modules of subsystem 201may include its own processing circuitry and/or memory. Alternativelyeach module of subsystem 201 may share processing circuitry and/ormemory with any other module of subsystem 201 and/or processor 102and/or memory 104 of system 100.

While there have been described systems, methods, and computer-readablemedia for generating computer-mediated reality display data based onuser instantaneous motion data, it is to be understood that many changesmay be made therein without departing from the spirit and scope of thesubject matter described herein in any way. Insubstantial changes fromthe claimed subject matter as viewed by a person with ordinary skill inthe art, now known or later devised, are expressly contemplated as beingequivalently within the scope of the claims. Therefore, obvioussubstitutions now or later known to one with ordinary skill in the artare defined to be within the scope of the defined elements.

Therefore, those skilled in the art will appreciate that the inventioncan be practiced by other than the described embodiments, which arepresented for purposes of illustration rather than of limitation.

What is claimed is:
 1. A computer-mediated reality system comprising: atleast one sensor; a mediated reality data source; and a mediated realitydisplay generator that generates displayable mediated reality scene databased on: current reality data of the system from the at least onesensor; mediated reality data from the mediated reality data source; andinstantaneous motion data of the system from the at least one sensor,wherein the mediated reality display generator generates the displayablemediated reality scene data by: generating displayable mediated realityframe data based on the current reality data and the mediated realitydata; and selecting a portion of the displayable mediated reality framedata as the displayable mediated reality scene data based on theinstantaneous motion data.
 2. The system of claim 1, wherein: thecurrent reality data of the system is provided to the mediated realitydisplay generator from a first sensor of the at least one sensor; andthe instantaneous motion data of the system is provided to the mediatedreality display generator from a second sensor of the at least onesensor.
 3. The system of claim 1, wherein a center of the selectedportion is offset from a center of the displayable mediated realityframe data based on the instantaneous motion data.
 4. The system ofclaim 1, wherein the current reality data of the system is indicative ofthe current location of the system.
 5. The system of claim 4, wherein:the current reality data of the system is provided to the mediatedreality display generator from a first sensor of the at least onesensor; the first sensor comprises a global positioning system sensor.6. The system of claim 4, wherein the mediated reality data of thesystem is indicative of information that is both related to the currentlocation of the system and useful to a user of the system.
 7. The systemof claim 1, wherein the current reality data of the system is indicativeof the current orientation of the system.
 8. The system of claim 7,wherein: the current reality data of the system is provided to themediated reality display generator from a first sensor of the at leastone sensor; the first sensor comprises at least one of a compass sensorand a gyroscope sensor.
 9. The system of claim 1, wherein: theinstantaneous motion data of the system is provided to the mediatedreality display generator from a first sensor of the at least onesensor; the first sensor comprises at least one accelerometer sensor.10. A method comprising: determining a current reality of acomputer-mediated reality display system; determining a currentinstantaneous motion of the computer-mediated reality display system;and generating displayable mediated reality scene data based on thedetermined current reality and the determined current instantaneousmotion, wherein the generating the displayable mediated reality scenedata comprises: generating displayable mediated reality frame data basedon the determined current reality; and selecting a portion of thedisplayable mediated reality frame data as the displayable mediatedreality scene data based on the determined current instantaneous motion.11. The method of claim 10, wherein a center of the selected portion isoffset from a center of the displayable mediated reality frame databased on the determined current instantaneous motion.
 12. The method ofclaim 10, wherein: the determining the current reality of thecomputer-mediated reality display system comprises determining a firstcurrent reality of the computer-mediated reality display system at afirst time; the determining the current instantaneous motion of thecomputer-mediated reality display system comprises determining a firstcurrent instantaneous motion of the computer-mediated reality displaysystem at the first time; the generating the displayable mediatedreality scene data based on the determined current reality and thedetermined current instantaneous motion comprises generating firstdisplayable mediated reality scene data based on the determined firstcurrent reality and the determined first current instantaneous motion;and the method further comprises: determining a second current realityof the computer-mediated reality display system at a second time afterthe first time; determining a second current instantaneous motion of thecomputer-mediated reality display system at the second time; andgenerating second displayable mediated reality scene data based on thedetermined second current reality and the determined second currentinstantaneous motion.
 13. The method of claim 12, wherein: thegenerating the first displayable mediated reality scene data comprises:identifying first displayable mediated reality frame data based on thedetermined first current reality; and selecting a field of view portionof the identified first displayable mediated reality frame data as thefirst displayable mediated reality scene data; and the position of theselected field of view portion of the identified first displayablemediated reality frame data with respect to the entirety of theidentified first displayable mediated reality frame data comprises afirst position that is based on the determined first currentinstantaneous motion.
 14. The method of claim 13, wherein: thegenerating the second displayable mediated reality scene data comprises:identifying second displayable mediated reality frame data based on thedetermined second current reality; and selecting a field of view portionof the identified second displayable mediated reality frame data as thesecond displayable mediated reality scene data; the position of theselected field of view portion of the identified second displayablemediated reality frame data with respect to the entirety of theidentified second displayable mediated reality frame data comprises asecond position that is based on the determined second currentinstantaneous motion; and the first position is different than thesecond position.
 15. The method of claim 14, wherein the first positionis different than the second position with respect to two axes.
 16. Anon-transitory computer-readable medium comprising computer-readableinstructions recorded thereon for: displaying mediated reality scenedata; determining a current reality of a computer-mediated realitydisplay system; determining a current instantaneous motion of thecomputer-mediated reality display system; and virtually altering theframe rate of the displayed mediated reality scene data by moving thefield of view of the mediated reality scene data with respect to thedetermined current reality of the system based on the determined currentinstantaneous motion.
 17. The non-transitory computer-readable medium ofclaim 16, wherein the determined current instantaneous motion comprisesa derivative of the current acceleration of the computer-mediatedreality display system.
 18. The non-transitory computer-readable mediumof claim 17, wherein the moving comprises: moving the field of view withrespect to the determined current reality of the system in a firstparticular direction when the current acceleration is positive in thefirst particular direction; and moving the field of view with respect tothe determined current reality of the system in a second particulardirection opposite to the first particular direction when the currentacceleration is negative in the first particular direction.